

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>ElasticSearch 同步模块 &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />

  
  
    <link rel="shortcut icon" href="../../_static/favicon.ico"/>
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/jquery.js"></script>
        <script src="../../_static/underscore.js"></script>
        <script src="../../_static/doctools.js"></script>
    
    <script type="text/javascript" src="../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../genindex/" />
    <link rel="search" title="Search" href="../../search/" />
    <link rel="next" title="云同步模块" href="../cloud-sync-module/" />
    <link rel="prev" title="同步模块" href="../sync-modules/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../../" class="icon icon-home"></a> &raquo;</li>
        
          <li><a href="../">Ceph 对象网关</a> &raquo;</li>
        
          <li><a href="../sync-modules/">同步模块</a> &raquo;</li>
        
      <li>ElasticSearch 同步模块</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
          
            <a href="../../_sources/radosgw/elastic-sync-module.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../">
          

          
            
            <img src="../../_static/logo.png" class="logo" alt="Logo"/>
          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../start/intro/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../">Ceph 对象网关</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../frontends/">HTTP 前端</a></li>
<li class="toctree-l2"><a class="reference internal" href="../placement/">存储池归置与存储类</a></li>
<li class="toctree-l2"><a class="reference internal" href="../multisite/">多站配置</a></li>
<li class="toctree-l2"><a class="reference internal" href="../multisite-sync-policy/">多站同步策略配置</a></li>
<li class="toctree-l2"><a class="reference internal" href="../pools/">存储池的配置</a></li>
<li class="toctree-l2"><a class="reference internal" href="../config-ref/">配置参考</a></li>
<li class="toctree-l2"><a class="reference internal" href="../admin/">管理指南</a></li>
<li class="toctree-l2"><a class="reference internal" href="../s3/">S3 API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rgw-cache/">Data caching and CDN</a></li>
<li class="toctree-l2"><a class="reference internal" href="../swift/">Swift API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../adminops/">管理操作 API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../api/">Python 接口</a></li>
<li class="toctree-l2"><a class="reference internal" href="../nfs/">通过 NFS 导出</a></li>
<li class="toctree-l2"><a class="reference internal" href="../keystone/">与 OpenStack Keystone 对接</a></li>
<li class="toctree-l2"><a class="reference internal" href="../barbican/">与 OpenStack Barbican 对接</a></li>
<li class="toctree-l2"><a class="reference internal" href="../vault/">与 HashiCorp Vault 对接</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kmip/">KMIP Integration</a></li>
<li class="toctree-l2"><a class="reference internal" href="../opa/">与 Open Policy Agent 对接</a></li>
<li class="toctree-l2"><a class="reference internal" href="../multitenancy/">多租户</a></li>
<li class="toctree-l2"><a class="reference internal" href="../compression/">压缩</a></li>
<li class="toctree-l2"><a class="reference internal" href="../ldap-auth/">LDAP 认证</a></li>
<li class="toctree-l2"><a class="reference internal" href="../encryption/">服务器端加密</a></li>
<li class="toctree-l2"><a class="reference internal" href="../bucketpolicy/">桶策略</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dynamicresharding/">动态的桶索引重分片</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mfa/">多因子认证</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../sync-modules/">同步模块</a><ul class="current">
<li class="toctree-l3 current"><a class="current reference internal" href="#">ElasticSearch 同步模块</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#elasticsearch-tier-type-configurables">ElasticSearch tier type configurables</a></li>
<li class="toctree-l4"><a class="reference internal" href="#end-user-metadata-queries">End user metadata queries</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../cloud-sync-module/">Cloud 同步模块</a></li>
<li class="toctree-l3"><a class="reference internal" href="../pubsub-module/">PubSub 模块</a></li>
<li class="toctree-l3"><a class="reference internal" href="../archive-sync-module/">Archive 同步模块</a></li>
<li class="toctree-l3"><a class="reference internal" href="../sync-modules/#id2">需求和假设</a></li>
<li class="toctree-l3"><a class="reference internal" href="../sync-modules/#id3">同步模块的配置</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../notifications/">Bucket Notifications</a></li>
<li class="toctree-l2"><a class="reference internal" href="../layout/">RADOS 中的数据布局</a></li>
<li class="toctree-l2"><a class="reference internal" href="../STS/">STS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../STSLite/">STS Lite</a></li>
<li class="toctree-l2"><a class="reference internal" href="../keycloak/">Keycloak</a></li>
<li class="toctree-l2"><a class="reference internal" href="../role/">Role</a></li>
<li class="toctree-l2"><a class="reference internal" href="../session-tags/">Session Tags</a></li>
<li class="toctree-l2"><a class="reference internal" href="../orphans/">Orphan List and Associated Tooliing</a></li>
<li class="toctree-l2"><a class="reference internal" href="../oidc/">OpenID Connect Provider</a></li>
<li class="toctree-l2"><a class="reference internal" href="../troubleshooting/">故障排除</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../man/8/radosgw/">radosgw 手册页</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../man/8/radosgw-admin/">radosgw-admin 手册页</a></li>
<li class="toctree-l2"><a class="reference internal" href="../qat-accel/">使用 QAT 为加密和压缩提速</a></li>
<li class="toctree-l2"><a class="reference internal" href="../s3select/">S3-select</a></li>
<li class="toctree-l2"><a class="reference internal" href="../lua-scripting/">Lua Scripting</a></li>
<li class="toctree-l2"><a class="reference internal" href="../d3n_datacache/">D3N Data Cache</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cloud-transition/">Cloud Transition</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev/developer_guide/">开发者指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev/internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <div class="section" id="elasticsearch">
<h1>ElasticSearch 同步模块<a class="headerlink" href="#elasticsearch" title="Permalink to this headline">¶</a></h1>
<div class="versionadded">
<p><span class="versionmodified added">New in version Kraken.</span></p>
</div>
<p>这个同步模块把其它域的元数据写入 <a class="reference external" href="https://github.com/elastic/elasticsearch">ElasticSearch</a> 。截止到
luminous ，我们存储在 ElasticSearch 中的是由很多数据字段组成的一个 json 。</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">{</span>
     <span class="s2">&quot;_index&quot;</span> <span class="p">:</span> <span class="s2">&quot;rgw-gold-ee5863d6&quot;</span><span class="p">,</span>
     <span class="s2">&quot;_type&quot;</span> <span class="p">:</span> <span class="s2">&quot;object&quot;</span><span class="p">,</span>
     <span class="s2">&quot;_id&quot;</span> <span class="p">:</span> <span class="s2">&quot;34137443-8592-48d9-8ca7-160255d52ade.34137.1:object1:null&quot;</span><span class="p">,</span>
     <span class="s2">&quot;_score&quot;</span> <span class="p">:</span> <span class="mf">1.0</span><span class="p">,</span>
     <span class="s2">&quot;_source&quot;</span> <span class="p">:</span> <span class="p">{</span>
       <span class="s2">&quot;bucket&quot;</span> <span class="p">:</span> <span class="s2">&quot;testbucket123&quot;</span><span class="p">,</span>
       <span class="s2">&quot;name&quot;</span> <span class="p">:</span> <span class="s2">&quot;object1&quot;</span><span class="p">,</span>
       <span class="s2">&quot;instance&quot;</span> <span class="p">:</span> <span class="s2">&quot;null&quot;</span><span class="p">,</span>
       <span class="s2">&quot;versioned_epoch&quot;</span> <span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
       <span class="s2">&quot;owner&quot;</span> <span class="p">:</span> <span class="p">{</span>
         <span class="s2">&quot;id&quot;</span> <span class="p">:</span> <span class="s2">&quot;user1&quot;</span><span class="p">,</span>
         <span class="s2">&quot;display_name&quot;</span> <span class="p">:</span> <span class="s2">&quot;user1&quot;</span>
       <span class="p">},</span>
       <span class="s2">&quot;permissions&quot;</span> <span class="p">:</span> <span class="p">[</span>
         <span class="s2">&quot;user1&quot;</span>
       <span class="p">],</span>
       <span class="s2">&quot;meta&quot;</span> <span class="p">:</span> <span class="p">{</span>
         <span class="s2">&quot;size&quot;</span> <span class="p">:</span> <span class="mi">712354</span><span class="p">,</span>
         <span class="s2">&quot;mtime&quot;</span> <span class="p">:</span> <span class="s2">&quot;2017-05-04T12:54:16.462Z&quot;</span><span class="p">,</span>
         <span class="s2">&quot;etag&quot;</span> <span class="p">:</span> <span class="s2">&quot;7ac66c0f148de9519b8bd264312c4d64&quot;</span>
       <span class="p">}</span>
     <span class="p">}</span>
   <span class="p">}</span>
</pre></div>
</div>
<div class="section" id="elasticsearch-tier-type-configurables">
<h2>ElasticSearch tier type configurables<a class="headerlink" href="#elasticsearch-tier-type-configurables" title="Permalink to this headline">¶</a></h2>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">endpoint</span></code></p></li>
</ul>
<p>Specifies the Elasticsearch server endpoint to access</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">num_shards</span></code> (integer)</p></li>
</ul>
<p>The number of shards that Elasticsearch will be configured with on
data sync initialization. Note that this cannot be changed after init.
Any change here requires rebuild of the Elasticsearch index and reinit
of the data sync process.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">num_replicas</span></code> (integer)</p></li>
</ul>
<p>The number of the replicas that Elasticsearch will be configured with
on data sync initialization.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">explicit_custom_meta</span></code> (true | false)</p></li>
</ul>
<p>Specifies whether all user custom metadata will be indexed, or whether
user will need to configure (at the bucket level) what custom
metadata entries should be indexed. This is false by default</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">index_buckets_list</span></code> (comma separated list of strings)</p></li>
</ul>
<p>If empty, all buckets will be indexed. Otherwise, only buckets
specified here will be indexed. It is possible to provide bucket
prefixes (e.g., foo*), or bucket suffixes (e.g., *bar).</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">approved_owners_list</span></code> (comma separated list of strings)</p></li>
</ul>
<p>If empty, buckets of all owners will be indexed (subject to other
restrictions), otherwise, only buckets owned by specified owners will
be indexed. Suffixes and prefixes can also be provided.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">override_index_path</span></code> (string)</p></li>
</ul>
<p>if not empty, this string will be used as the elasticsearch index
path. Otherwise the index path will be determined and generated on
sync initialization.</p>
</div>
<div class="section" id="end-user-metadata-queries">
<h2>End user metadata queries<a class="headerlink" href="#end-user-metadata-queries" title="Permalink to this headline">¶</a></h2>
<div class="versionadded">
<p><span class="versionmodified added">New in version Luminous.</span></p>
</div>
<p>Since the ElasticSearch cluster now stores object metadata, it is important that
the ElasticSearch endpoint is not exposed to the public and only accessible to
the cluster administrators. For exposing metadata queries to the end user itself
this poses a problem since we’d want the user to only query their metadata and
not of any other users, this would require the ElasticSearch cluster to
authenticate users in a way similar to RGW does which poses a problem.</p>
<p>As of Luminous RGW in the metadata master zone can now service end user
requests. This allows for not exposing the elasticsearch endpoint in public and
also solves the authentication and authorization problem since RGW itself can
authenticate the end user requests. For this purpose RGW introduces a new query
in the bucket APIs that can service elasticsearch requests. All these requests
must be sent to the metadata master zone.</p>
<div class="section" id="id1">
<h3>语法<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h3>
<div class="section" id="get-an-elasticsearch-query">
<h4>Get an elasticsearch query<a class="headerlink" href="#get-an-elasticsearch-query" title="Permalink to this headline">¶</a></h4>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>GET /{bucket}?query={query-expr}
</pre></div>
</div>
<dl class="simple">
<dt>request params:</dt><dd><ul class="simple">
<li><p>max-keys: max number of entries to return</p></li>
<li><p>marker: pagination marker</p></li>
</ul>
</dd>
</dl>
<p><code class="docutils literal notranslate"><span class="pre">expression</span> <span class="pre">:=</span> <span class="pre">[(]&lt;arg&gt;</span> <span class="pre">&lt;op&gt;</span> <span class="pre">&lt;value&gt;</span> <span class="pre">[)][&lt;and|or&gt;</span> <span class="pre">...]</span></code></p>
<p>op is one of the following:
&lt;, &lt;=, ==, &gt;=, &gt;</p>
<p>For example</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>GET /?query=name==foo
</pre></div>
</div>
<p>Will return all the indexed keys that user has read permission to, and
are named ‘foo’.</p>
<p>The output will be a list of keys in XML that is similar to the S3
list buckets response.</p>
</div>
<div class="section" id="id2">
<h4>配置定制的元数据字段<a class="headerlink" href="#id2" title="Permalink to this headline">¶</a></h4>
<p>Define which custom metadata entries should be indexed (under the
specified bucket), and what are the types of these keys. If explicit
custom metadata indexing is configured, this is needed so that rgw
will index the specified custom metadata values. Otherwise it is
needed in cases where the indexed metadata keys are of a type other
than string.</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>POST /{bucket}?mdsearch
x-amz-meta-search: &lt;key [; type]&gt; [, ...]
</pre></div>
</div>
<p>Multiple metadata fields must be comma separated, a type can be forced for a
field with a <cite>;</cite>. The currently allowed types are string(default), integer and
date</p>
<p>eg. if you want to index a custom object metadata x-amz-meta-year as int,
x-amz-meta-date as type date and x-amz-meta-title as string, you’d do</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>POST /mybooks?mdsearch
x-amz-meta-search: x-amz-meta-year;int, x-amz-meta-release-date;date, x-amz-meta-title;string
</pre></div>
</div>
</div>
<div class="section" id="id3">
<h4>删除定制的元数据配置<a class="headerlink" href="#id3" title="Permalink to this headline">¶</a></h4>
<p>删除定制的元数据桶配置。</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>DELETE /&lt;bucket&gt;?mdsearch
</pre></div>
</div>
</div>
<div class="section" id="id4">
<h4>查看定制的元数据配置<a class="headerlink" href="#id4" title="Permalink to this headline">¶</a></h4>
<p>检出定制的元数据桶配置。</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>GET /&lt;bucket&gt;?mdsearch
</pre></div>
</div>
</div>
</div>
</div>
</div>



           </div>
           
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="../cloud-sync-module/" class="btn btn-neutral float-right" title="云同步模块" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
        <a href="../sync-modules/" class="btn btn-neutral float-left" title="同步模块" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

    </p>
  </div> 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>